We study the effect on the cosmic microwave background (CMB) anisotropy and large scale structure (LSS) power spectrum of a scattering interaction between cold dark matter and baryons. This scattering alters the CMB anisotropy and LSS spectrum through momentum transfer between the cold dark matter particles and the baryons. We find that current CMB observations can put an upper limit on the scattering cross section which is comparable with or slightly stronger than previous disk heating constraints at masses greater than 1 GeV, and much stronger at smaller masses. When large-scale structure constraints are added to the CMB limits, our constraint is more stringent than this previous limit at all masses. In particular, a dark matter-baryon scattering cross section comparable to the ``Spergel-Steinhardt'' cross section is ruled out for dark matter mass greater than 1 GeV.